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(54) Computerized test preparation system employing individually tailored diagnostics and 
remediation 



(57) A computerized test preparation system and 
methods are disclosed which assist an examinee to in- 
crease his learning and improve his performance on 
standardized academic or applied aptitude and achieve- 
ment exams. Performance feedback information is pro- 
vided to a user, including conventional information such 
as number of items correct and time elapsed, pinpoints 
substantive areas of a particular exam in which the user 
is either weak or strong, and offers diagnoses of, and 
remediation tailored to, a user's personal difficulties. By 
assessing a user's responses to given exam questions. 



through various distractor error patterns manifested in 
the incorrect answers to multiple-choice questions, or 
through coded categories of responses in the case of 
user responses to constrained open-ended items, a pre- 
liminary diagnosis of a user's deficits in knowledge and/ 
ortest taking skill is made. This preliminary diagnosis is 
subsequently refined utilizing additional examination 
questions. The refined diagnosis is then utilized to offer 
recommendations to an individual user to ameliorate the 
user's manifested error patterns. A user's response to 
this remediation is confirmed and reinforced using sub- 
sequent remediation activities. 
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Description 

Field of the Invention 

5 [0001] The present invention relates to a connputerized, or other nnachine based test preparation systenn, and nnore 
particularly, to a nnethod and apparatus for enhancing learning and innproving exanninee scores on standardized exanns 
through the use of individually tailored diagnostics and rennediation. 

Background of the Invention 

10 

The proliferation of "high-stakes" examinations and conventional test preparation methods 

[0002] "High-stakes" exanninations are very connnnon today. Typically they are tinne-based exams testing a set of 
predeternnined subject areas. A nunnber of these examinations, such as the Scholastic Aptitude Test ("SAT") series of 

15 examinations and others like it (e.g., MCAT, LSAT, ACT, GRE, GED, GLEP, BAR exam, DMV exams), have been labeled 
as "high-stakes" testing. In such "high-stakes" tests, the primary objective is the placement of an examinee on a latent 
trait or ability dimension, for a variety of purposes such as selection and placement (I.e., SAT, l\/ICAT or LSAT), or 
certification (i.e., GED and DMV exams). Most of these tests include items from a variety of scholastic domains (e.g., 
SAT: verbal, mathematics; LSAT: logical reasoning, reading comprehension, verbal) that are arranged in a formal struc- 

20 ture. The test items are chosen and developed by the test makers so as to "reliably" place examinees on the latent 
dimension of interest to the examiner and consumers of the standardized scores from such exams. 
[0003] One factor which is thought to be capable of influencing examinee performance on these tests is coaching, 
or formal test preparation efforts. Because of the proliferation of these "high-stakes" examinations, an entire test prep- 
aration industry has arisen to help prepare examinees and improve their scores on these exams. Offerings include 

25 classroom-based tutoring, stand-alone printed publications, and computer-based materials (e.g., disk, CD-ROM, in- 
ternet). All of these offerings claim to be able to increase an examinee's score on the particular standardized exams 
to which they are directed. 

[0004] The most conventional test preparation offerings have been traditionally represented by such organizations 
as Kaplan Learning and The Princeton Review study centers, or self-study methods based on printed test preparation 
30 texts such as 10 Real SATS, and Gruber's Complete Preparation for the New SAT: Eighth Edition. Through the use of 
such methods, examinee score Increases have been modest, generally resulting in score increases on the order of 
1/5th of a standard deviation.. 

[0005] More recently, computer based exam preparation materials have been developed and offered including The 
College Board's One-on-One with the SAT, The Princeton Review's Inside the SAT, ACT & PSAT, published by The 
35 Learning Company, and The Crash Course for SAT, PSAT & ACT, published by ARCO Publishing. Additionally, some 
of the testing centers mentioned above have begun offering computerized training materials generally corresponding 
to their traditional classroom based approach. 

[0006] Common characteristics of these computerized offerings include: (i) presentation of timed "sample exams" 

and practice exams, (ii) scoring of responses from these exams, (iii) some question-specific feedback (e.g., response 
40 chosen, correct answer, brief explanation), and (iv) general test-taking tips (e.g., pacing, skipping questions). Features 
which differentiate these offerings include: (i) the use or non-use of audio and/or graphics, (ii) the ability to mark items 
to be skipped and returned to, (iii) feedback of a study plan based upon the results of a "sample exam", and (iv) the 
provision of explanations for each of the response alternatives for each item. 

[0007] Several of these computerized offerings have been distributed over the internet. Some of the web sites offering 
45 exam preparation and review include: (i) Score.Kaplan.com (based on materials offered by Kaplan Learning), (ii) Re- 
view.com, (iii) Testprep.com, (iv) ACTIve Prep atAct.org, (v) powerprep.com, and (vi) Novanet.com (based on materials 
offered by The Princeton Review). A review of these web sites as they existed in November, 2000 revealed variations 
in complexity from "page-turners" to relatively complete implementations of the printed volumes on which some of them 
are based. In general though, they reflect the same range of complexity and operation as found in the other compu- 
te terized and CD-ROM offerings discussed. 

[0008] Several of the web-based offerings also provide "sample exams" which can be taken by the user. Information 
is generally fed back to the user of such offerings in the form of raw and scaled scores. In some cases, the feedback 
may also include a re-presentation of the exam items, the indication of the user's response and the correct choice, and 
an explanation of why the correct answer is correct and why each of the alternatives are wrong. While responses to 
55 the "sample exams" in some cases provide the basis for "diagnostic" feedback, the diagnosis in this context is defined 
from a conventional testing perspective and is determined merely by the number of incorrect answers rather than the 
types of incorrect answers. Thus, a study plan, or diagnosis, if provided, is usually based upon the user's distribution 
of scores across the various sections of the examination and results in a simplistic recommendation of remediation, 
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such as the need to review geometric principles or increase vocabulary. 
Recent Development in Cognitive Diagnostic Assessments 

5 [0009] Educators and researchers, influenced by recent developments in cognitive psychology and societal concerns 
regarding the influence of testing on equality of education, have sought testing instruments that would reveal the mech- 
anisms, structures and processes that are activated when an examinee takes a test, and thus, would inform the in- 
structional process. Conventional tests, while adequately serving as selection and/or placement instruments, are not 
well suited for determining a course of instruction or for identifying the source of problem-solving errors. 

10 [0010] Acategory of testing called cognitively diagnostic assessment ("CDA") or dynamic testing has been developed 
which may provide a basis for individualized instruction for each examinee in a domain of interest. Such tests are based 
upon cognitive theories of learning, and as such, are not concerned with the representative sampling of items from a 
content domain (such as algebraic equations), but rather, with the examinee's knowledge and application of cognitive 
attributes which are thought to be required or not required to adequately solve a given problem. CDA testing provides 

15 infonnation regarding the strategies that examinees use to attack problems, relationships they perceive among con- 
cepts, and principles they understand in a domain. The goal of these testing methods is to determine, on the basis of 
a simple test, what the strengths and weaknesses of an examinee are, relative to a specified list of cognitive attributes 
of interest to the teacher and the tester. 

[0011] CDA-type tests are typically built around an attribute by item matrix (i.e., a Q-matrix). Thus, for an examinee 
20 to solve a given problem, it is assumed that they have knowledge of, and the ability to apply, one or more cognitive 
attributes related to the item or problem. The failure of an examinee to solve a problem is then attributed to the absence 
of a requisite cognitive attribute or to a lack of skill in its application. 

[0012] The major difficulty experienced with most CDA tests is one of numerosity - the number of possible sources 
of error grows exponentially as the number of attributes and the number of items increase. For example, some attempts 

25 by researchers to form a Q-matrix for 60 items on the SAT math test yielded more than 3,000 prototypical error patterns. 
Other researchers developed models containing only 4 attributes - strategy, completeness, positivity and slips - which 
were proposed to be evocative of properties that could be uses in developing and interpreting diagnostic assessment 
tests. An evaluation of all of these models revealed that such a small universe of attributes could not adequately capture 
the test takers' cognitive deficiencies, while large attribute approaches were unlikely to provide a practical means of 

30 cognitive diagnostic assessment based upon simple testing. 

[0013] Improvements in testing have been made possible by advancements in computer technology as well as ad- 
vancements in cognitive theory. However, because there are always more ways to get an item wrong than right on a 
multiple-choice exam (i.e., on a typical multiple-choice question, there is only one correct answer and 3 or 4 distractors 
or incorrect options), or even more so, with regard to open-response, "fill-in-the-blank" questions, the specification of 

35 the cognitive model space remains a difficult task. Currently available cognitive diagnostic assessment programs are 
not able to handle the complexity of SAT-type examination questions. 

Scoring of Multiple Choice and Constructed Response Examination Items 

40 [0014] Multiple choice ("MC") tests are composed of items having two sub-parts - a stem representing the question 
and a series of response alternatives, one of which is the correct response. It is the presence of the response alterna- 
tives which differentiates MC test items from constructed response items which contain no response alternatives and 
require an examinee to self-generate a response. MC tests are typically scored by comparing the examinee's response 
to an item against a key that contains the correct answers. This is dichotomous scoring, 0 or 1 , the answer being either 

45 correct or incorrect. Polychotomous scoring methods assign weights to each of the response options, with the correct 
response being given the largest weight. In practice, the two scoring methods yield highly correlated sets of test scores. 
Polychotomous scoring methods utilize more of the information available in a set of incorrect answers or distractors, 
although solely in service of the conventional testing purpose of rank ordering examinees, rather than for instruction 
or remediation. 

50 [0015] Under either method, the set of response alternatives (the incorrect alternatives being known as distractors 
or foils) assumes considerable importance. If the distractors don't work, the test item becomes unreliable, and the 
interpretation of the scores becomes meaningless. Traditional methods of test construction have focused on the se- 
lection of distractors that are thought to yield some information about the latent trait being evaluated and on the elim- 
ination of non-working distractors. Conventionally, information derived from incorrect responses to an exam question 

55 is solely used by test developers to indicate that the question needs improvement, either in the wording of the stem or 
in the specification of the response alternatives. Nonetheless, additional useful information about the examinee can 
be captured from these incorrect answers. Researchers have observed that classification of response option choice 
according to type of error could be utilized for diagnostic purposes. Nevertheless, significant attention has not been 
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directed toward developing IVIC tests in wliich tine response alternatives are scored diagnostically for the benefit of the 
exanninee and exanniner. 

[0016] Constructed response items, such as short answer or essay questions, typically require a person knowledge- 
able in the donnain being tested to score such response itenns. Constrained constructed response items, such as the 
5 grid-in items on SAT-type examinations, may now be computer scored as the software programs are capable of ac- 
cepting a range of responses as being correct. The scoring routines employed for a majority of high stakes examinations 
are still designed to yield scores based on a binary correct/incorrect coding of responses. Programs for the scoring of 
responses to extended essay questions are still in the investigatory stage. 

10 Disadvantages of the Prior Art 

[0017] Known methods of preparing examinees for "high-stakes" exams are costly; fail to hold the interest of the 

examinee, and are inefficient and inconvenient. Furthermore, these current methods generally provide a low return on 
an examinee's investment, both financially and mentally. Reviews of research on admissions test coaching indicated 

15 that score increases are on the order of 1/5th of a standard deviation. 

[0018] More importantly, since the current methods of test preparation remain wedded to the traditional concept of 
ranking each examinee against another on a latent dimension using scaled scores, the failure of an examinee to achieve 
a "satisfactory score" (as either defined by a school or other agency, or self-defined) results in a course of remediation 
limited to simple recommendations of more practice in a particular area; a method of remediation which is only weakly, 

20 if at all, informed by the test-taking experience. Feedback to the user which is based on such conventional test con- 
siderations does little to facilitate learning or improvement in knowledge in the domain of study. 
[001 9] Known test preparation systems also do not provide for the cognitive diagnosis of test-taking and/or content- 
related problems. Recommendations for remediation are based on the overall frequency of wrong answers in specific 
domains of a test, rather than on the frequency of specific types of wrong answers. 

25 [0020] Furthermore, current systems do not permit the user to adapt the study program so that it is maximally effective 
for the particular user. Users who are visually oriented and learn most effectively from graphical presentations are 
provided no different manner of instruction than those users who are aurally oriented and who would benefit more from 
spoken explanations. 

[0021] Finally, while mentoring, or one-on-one tutoring, remedy many of the shortcomings in these conventional test 
30 preparation methods, and while use of these methods is perhaps the most effective manner of diagnosing learning 
difficulties and effecting remedial action, individual mentoring is very costly and qualified mentors are limited in numbers 
and availability. Thus, once again, neither of these options are viable solutions for facilitating improvement in examinee 
scores on standardized tests in a commercially reasonable manner. 

35 Objects and Summary of the Invention 

[0022] The invention provides an apparatus and method for enhancing learning and improving examinee test scores 
on standardized tests using cognitive diagnostic principles of diagnosis and remediation. IVIore specifically, the invention 
provides a comprehensive, self-contained system for assessing and preliminarily diagnosing patterns of examinee 

40 errors through the use of data from the incorrect response alternatives (distractors) presented in each multiple-choice 
exam question or presented in response to constrained open-ended exam questions, confirming the preliminary diag- 
nosis, if necessary, through the use of subsequent examination, offering remediation based upon the diagnosed error 
patterns, and reinforcing this remediation through skill development exercises, in order to increase an examinee's 
learning and level of performance on standardized tests. According to one embodiment, the invention utilizes informa- 

45 tion inherent in the distractors in standardized multiple-choice tests. According to another embodiment, the invention 
utilizes information provided as responses to constrained open-ended exam questions, in which the stem of such 
questions mirrors those employed in standardized multiple-choice tests. A system incorporating either of these em- 
bodiments does not require the creation of new test questions or responses. According to another embodiment, specific 
distractors are included in evaluation examinations that provide additional insight and information in identifying problem 

50 solving deficiencies. According to another embodiment of the invention, coded categories of responses that correspond 
to user generated response items are employed to provide infomnation analogous to that provided by the distractors 
in standardized multiple-choice tests. 

[0023] A system designed according to one embodiment of the invention incorporates several different program 
components. Those components may include a user interface, a test generator, a diagnostic scoring component, and 
55 a remediation component. The user interface manages a user's interaction with the system, requests and stores various 
personal infomriation with respect to the user, and allows the system to be specifically tailored to the individual user. 
The test generation component compiles and formats various types of examinations for provision to the user, such as 
diagnostic sample tests, non-diagnostic test-taking strategy tests, and basic skill tests, and presents the examinations 
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to the user for completion , storing a variety of information witli regard to the user's responses to the exam . The diagnostic 
component assesses and diagnoses (both preliminarily and through a more informed manner) a user's error patterns 
in connection with the tests generated from the test generation component. The remediation component employs 
diagnoses from the diagnostic component to recommend remedial activities for improving examinee test performance 

5 and scores. The remediation component additionally contains a number of features that, in connection with the user 
interface component, allow the system to be specifically tailored to an individual user. Such features include the des- 
ignation of materials for specific types of presentation, scrolling and bookmarking of materials, the presentation of 
difficulty meter levels, and the use of various multi-media features for presentation of remediation materials. 
[0024] The systems and methods according to one aspect of the invention identify patterns of errors in a user's 

10 choice of distractors contained within current standardized tests, and provide individually tailored remedial activities 
selected and based on such patterns. According to another aspect of the invention, test questions are developed which 
have stems and correct answers that are parallel to current standardized tests, but which have distractors that are 
designed to identify specific problem solving errors. According to yet another aspect of the invention, questions are 
developed independent of current standardized tests, and which have distractors designed to identify specific cognitive 

15 errors. Analysis of the selection of incorrect and correct answers is used to develop an individualized program of 
remediation. 

[0025] According to an embodiment of the invention wherein current published standardized tests are used to compile 
the exam questions by the test generation component, the content and format of a particulartest determines the overall 
number of distractor codes that are assigned. Since many standardized tests have been developed using a variety of 

20 item analysis techniques, the items and associated distractors which comprise the final versions of these tests are 
considered to be effective at assessing the examinee's knowledge of the content domain. Thus, a system according 
to the present invention needs only to determine the information value of the incorrect responses and to assign category 
codes that reflect the probable error type made by an examinee that chose the incorrect alternative for tests of this 
type. According to another embodiment wherein exam items are generated which are specifically tailored to assess a 

25 user's response to specified distractor codes, a more detailed range of codes can be assigned. 

[0026] Known test preparation systems do not provide for the diagnosis of error patterns that exist in an user's choice 
of incorrect response alternatives, and thus do not have the capability of recommending a course of remediation and/ 
or skill development on the basis of a user's having responded to a sample standardized test in which the response 
options were not only scored as correct or incorrect, but also in terms of the types of errors they represent. 

30 [0027] In light of the limitations of known tests preparation systems and methods, it is an object of the invention to 
provide a more efficient, convenient, and effective manner of enhancing learning and improving test scores for a variety 
of "high stakes" examinations. It is another object of the present invention to provide a test preparation system and 
method wherein an examinee's error patterns with respect to incorrect responses to exam questions are assessed and 
the examinee's cognitive deficiencies diagnosed, and, using this information, recommendations are made for remedial 

35 activities targeted to the individual examinee. 

[0028] Another object of the present invention is to provide a system and method for teaching "testwiseness" skills, 
or skills which incorporate the use of cues provided by the test itself, or which are obtained by knowledge of the pro- 
pensities of the test maker, to arrive at correct answer to exam questions without possessing an underlying knowledge 
of how and why a particular answer is correct. 

40 [0029] The various aspects of the invention discussed above may also be combined in various ways to produce 
additional advantages of the invention over known systems and methods. For example, the present invention may be 
used to provide for remedial training and/or skill development informed by the assessment of an individual examinee's 
error patterns, provided at a customer's site. The present invention may also provide for a flexible presentation of test 
contents and materials, in both visual and audio form, tailored to the unique needs of a particular examinee as chosen 

45 by the examinee himself .In addition, further objects and advantages afforded by the present invention will be apparent 
from the detailed description hereinbelow. 

Brief Description of tlie Drawings 

50 [0030] The appended drawings are presented to further describe the present invention and to assist in its under- 
standing through clarification of its various aspects. The features, characteristics and advantages discussed above, 
as well as other features, characteristics and advantages of the invention, will be apparent to a person of ordinary skill 
in the art upon consideration of the following detailed description, taken in conjunction with the accompanying drawings, 
in which: 

55 

Fig. 1 illustrates a block diagram of an embodiment of a diagnostic testing and remediation system according to 
the invention; 
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Fig. 2 illustrates a block diagram of the user interface component according to the invention; 

Fig. 3 illustrates a block diagram demonstrating the interaction between the user interface component, the reme- 
diation component and the test generation component according to the invention; 

5 

Fig. 4 illustrates a block diagram of the test generation component according to the invention; 

Fig. 5 illustrates a block diagram demonstrating the interaction between the test generation component and the 
diagnostic component according to the invention; 

10 

Fig. 6 illustrates a block diagram of the implementation of the diagnostic component according to the invention; 

Fig. 7 illustrates a block diagram of a further embodiment of the diagnostic component according to the invention, 
wherein the interaction between the diagnostic component and the remediation component according to the in- 
15 vention is illustrated; 

Fig. 8 illustrates a block diagram of the remediation component, wherein the interaction between the remediation 
component and the test generation component according to the invention is illustrated; and 

20 Fig. 9 illustrates a block diagram of the test generation component according to the invention. 

Detailed Description of the Invention 

[0031] Fig. 1 is a block diagram of an embodiment of a diagnostic testing and remediation system according to the 
25 present invention. An apparatus implementing the present invention may take multiple forms depending on the con- 
figuration of the components. The system includes a Computing Device 72 which maybe local (e.g., desktop) or remote 
(e.g., networked, internet, intranet), an Output Device 73, which may be a CRT, flat panel, and/or other similar display 
device or projector, an Input Device 74, which may take the form of a keyboard, pressure sensitive switch panel, voice 
recognition system, biometric sensor and/or other similar device, and a Memory/Persistent Storage System 75 which 
30 may be local, as in a diskette, DVD/CD-ROM, or hard drive, or remote. 

[0032] The Memory/Persistent Storage System 75 stores various additional components which can be utilized in 
implementing the system, such as the Operating System 76, Programming 77, User Demographics and Qualifications 

78, Test Item Pool 79, and Other Data 80. The Operating System 76 may include Windows/DOS, IVIAC-OS, Linux, and 
other like computer operating systems. The User Demographics and Qualifications Component 78 may include infor- 
ms mation about one or more examinees such as gender, age, grade level, prior test and test preparation experiences 

and the like. The system according to the invention may use the information in the User Demographics and Qualifica- 
tions Component 78 to make appropriate recommendations and personalize test item materials. 
[0033] The Test Item Pool 79 contains test items that are utilized to create a variety of multiple-choice tests. Each 
test item has descriptor codes associated with it that indicate, among other things, the item's domain, format, difficulty 
40 level or error category. The Other Data Component 80 may include an examinee's responses to diagnostic tests pre- 
viously administered by the system. The Programs Component 77 may include multiple components as illustrated in 
Figs. 2-5 which, according to one embodiment of the invention, substantially establish and implement the test prepa- 
ration system. 

[0034] The User Interface Component 1 , shown in detail in Fig. 2, manages the user interaction with the system and 
45 may perform such functions as credentialing and the display of various system options. The Test Generation Component 
1 0, shown in detail in Fig. 3, compiles and formats one or more tests by assembling test items from the Test Item Pool 

79. Various types of tests may be created by the Test Generation Component 1 0, including Diagnostic Sample Tests, 
which are formatted and timed as an analog of standardized paper and pencil tests. Test-taking Strategy Tests, which 
assess knowledge of test taking strategies and skill in their application (including testwiseness of the user), and Basic 

50 Skills Tests, which assess knowledge of basic facts in the domains being tested. Test items comprise a test question 
and an answer set including at least one correct and one incorrect answer. Alternatively, the tests generated by the 
Test Generation Component 1 0 include test items that comprise a question designed to elicit a constrained open-ended 
response from the test taker. Such items may consist of the stem portions of analogous MC items. Accordingly, in such 
circumstances, the answer set for these constrained open-ended items will contain at least one correct answer and 

55 multiple incorrect answers. 

[0035] The Testing Component 5 shown in Fig. 3, utilizes user choices on exam questions, information about the 
test items used to assemble the exam and user demographic data to develop one or more diagnoses to be used as a 
basis for making recommendations for remediation activities. When the tests generated by the Test Generation Com- 
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ponent 10 include test items tliat comprise questions designed to elicit a constrained opened-ended response from 
the test taker, information about the test items and user demographic data may still be employed to develop one or 
more diagnoses to be used as a basis for making recommendations for remediation activities. According to an em- 
bodiment where the constrained open-ended items consist of the stem portions of analogous MC items, the identical 
5 item information that is available for the MC items is available for the constrained open-ended items. The Remediation 
Component 47(a) shown in Fig. 7 employs diagnoses from the Diagnostic Component 5 and the Scoring Diagnostic 
Component 29, in conjunction with the Test Generation Component 10, to recommend and provide remedial activities, 
including recommended reading, suggested study materials, practice tests and the like. 

[0036] Figs. 2 and 3 are block diagrams of the User Interface Component according to the invention and illustrate 
10 the interaction between the User Interface Component 1 , the Remediation Component 47a and the Test Generation 
Component 10. In one embodiment, the User Interface Component 1 manages the user's interaction with the system 
and may perform such functions as credentialing and the display of various options offered by the system. The User 
Interface Component 1 begins by displaying an option 2 to the user to enter or exit the program. If the user chooses 
to enter the program, a Credentials Check 3 is completed to determine if the user is a former or new user and whether 
15 the Demographic Data Blank 4 still requires completion. This information is required so as to provide an appropriate 
or optimal listing of options 4a available to the user; to provide data to the Test Generation Component 10 so that 
appropriate tests may be constructed; and to provide data to the Remediation Component 47a so that appropriate 
weights may be applied to test generation and to the creation of feedback screens. 

[0037] Additionally, the Demographic Data Blank 4 requests information regarding age, gender, objectives, interests 
20 and the like so that certain test items may be personalized; and information regarding prior test-taking experience so 
that diagnostic recommendations 48 may be appropriately weighted. Once the Credential Checks has been completed, 
the user is presented with an Options List 4a. According to one embodiment, the user is allowed to select either Sample 
Diagnostic Tests 11 or go directly to the Remediation Component 47a for skills development. If the Sample Diagnostic 
Tests 1 1 are chosen, the Test Generation Component 1 0 provides the user with a selection of Sample Diagnostic Tests 
25 11 that are either computer based analogs of the standard paper and pencil versions, or tests which have been spe- 
cifically developed by the system to determine a user's errors in reasoning. The user is informed that once the Sample 
Diagnostic Tests 11 are completed, appropriate remediation activities will be recommended. 

[0038] A second Credential Check 6 permits the display of the appropriate set of options available to the user and 
the requirements for the various levels of diagnostic feedback. For those users who have not completed the requisite 
30 number of items on the Demographic Data Blank 4 to permit the full implementation of the diagnostic recommendation 
48 and feedback 61 processes, a Non-Credentialed User Options List 7 is available. For those users who have com- 
pleted a requisite number of items on the Demographic Data Blank 4 so that a full implementation of the diagnostic 
recommendation 48 and feedback 61 processes are possible, a separate Credentialed User Option List 8 is presented. 
[0039] Once the user encounters either Option List 7, 8, the user is provided a choice 9 of available Sample Diagnostic 
35 Tests 11 for selection. Once the user chooses from the available Sample Diagnostic Tests 11 , the Test Generation 
Component 10 controls the item selection, item formatting, and test presentation for the user selected Sample Diag- 
nostic Test 11 . According to one embodiment, the Sample Diagnostic Tests 11 are designed to diagnose of areas of 
content and test taking proficiency which require improvement. The Sample Diagnostic Tests 11 provide the basis for 
a preliminary diagnosis of types of user errors and for remedial recommendations. According to one embodiment, the 
40 Sample Diagnostic Tests 11 are taken under conditions simulating the "real-life" exams. 

[0040] According to another embodiment, the Sample Diagnostic Tests 1 1 are generated using specifically designed 
questions and response alternatives to implement the concept of adaptive remedation. According to this embodiment, 
for items of equal difficulty level and content, the response alternatives are varied to diagnose sensitivity to varied 
distractors, and thus, diagnose specified errors of reasoning. Sample Diagnostic Tests 11 , according to this embodi- 
es ment, do not rely on the limited set of distractors contained in the standard exams. Analysis of the selection of response 
alternatives allows a system according to this embodiment of the invention to accurately determine the user's level of 
knowledge for a particular content domain efficiently (i.e., using the least number of questions). According to a further 
embodiment, the test item pool is stratified by item difficulty level and the program algorithm selects subsequent items 
from the pool in accordance with the prior responses. According to a further embodiment where constrained open- 
50 ended items are employed which consist of the stem portions of analogous MC items, the difficulty level information 
that is available for the analogous MC items will also be available for the selection of constrained open-ended items 
for use in the Sample Diagnostic Tests 11 

[0041] Figs. 4 and 5 are block diagrams of the Test Generation Component 10 according to the invention and the 
interaction between the Test Generation Component 1 0 and the Diagnostic Component 5, 29 according to an embod- 
55 iment of the invention. As demonstrated in Fig. 4, after the user has selected the Sample Diagnostic Test 1 1 at step 9 
and the Sample Diagnostic Test 1 1 has been generated at step 1 0, a test section timer 1 2 is started and any applicable 
section instructions and sample problems are displayed 13. Once the user elects to continue, the first test item is 
displayed along with its response alternatives 14. The test item status is assessed as each test item is displayed, as 
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either new or old (previously displayed in a given session) 1 5 so as to appropriately set the itenn elapsed tinner 16, 17. 
For new itenns, the itenn elapsed tinner 1 6 is started at zero. For old itenns previously displayed, the itenn elapsed tinner 
17 continues fronn the tinne elapsed during the previous display. Once again, the user may choose fronn annong the 
response alternatives 18, including the choice to booknnark a particular itenn for later consideration if tinne pernnits. 
5 Once the user has nnade a choice 1 8, the elapsed itenn tinner stops 1 9, and the user's response and elapsed tinne are 
recorded 20. 

[0042] In one ennbodinnent, for each itenn connpleted, the progrann assesses 21 whether the nunnber of itenns conn- 
pleted "i" equals the number of items in a given sub-section "s", and if not, the program inquires if the time limit for a 
sub-section has been met 22. If the time limit 22 has not been exceeded, another item 14 is presented. If the number 

10 of items completed "i" equals the total number of items in the section or sub-section "s", or the section time limit 22 
has been met, the section timer is stopped 23. Once the section timer is stopped 23, the program assesses whether 
it is appropriate to permit an authorized break from the test 24. If an authorized break is permitted, the break timer 25 
times the duration of the authorized break from the test and notifies the user when to resume the test. 
[0043] In one embodiment, if no break was authorized or the authorized break is completed, the program assesses 

15 26 whether the number of sections in the exam completed "c" equals the total number of sections in the exam "t". In 
another embodiment, the program also assesses 27 whether the sum of the times for the completed sessions exceeds 
the total time allowed for the test. If the number of completed sections "c" equals the number of total sections "t" in the 
exam, or the total times for the completed sessions equals or exceeds the total allowable time, the program continues 
onto the Diagnostic Component 5, 29. If the sum of the times for the completed sections does not exceed the total time 

20 allowed for the test 27, and if the number of completed sections "c" does not equal the total number of sections in the 
exam "t", the next section of the exam is presented 28. 

[0044] Figs. 5, 6 and 7 are block diagram of the Scoring Diagnostic Component 29 and the interaction between the 
Scoring Diagnostic Component 29 and the Remediation Component 47a according to an embodiment of the invention. 
The Scoring Diagnostic Component 29 utilizes user responses to test items and test item information to formulate a 
25 preliminary diagnosis of content areas and test taking behavior needing remediation. This component also includes 
the scoring routines for the Sample Diagnostic Tests 1 1 . 

[0045] The program gathers the user responses and item information on an item-by-item basis for each sub-section 
"s" of a test 30. The user response to an item is compared to a scoring key 31 and if the two match, the total number 
of correct items for a sub-section "s" is incremented 32. This continues until all user responses are compared with all 

30 items in a particular sub-section "s" and the number of correct responses is totaled 32. Additionally, for each item within 
a given sub-section "s", the response alternative codes and item difficulty level codes are retrieved 33. User responses 
to each item are summed as to the values corresponding to the response alternative codes 38 and difficulty level 37. 
The program continues this assessment and scores additional items until the sub-section "s" is complete 34, 35. The 
program continues the scoring assessment for each sub-section "s" until the test is complete and no sub-sections "s" 

55 remain 34, 35. Once the test scoring is complete, the sub-section "s" scores are summed to determine the total test 
score "T" 36. Additionally, the averaged elapsed time is calculated for each sub-section of the sample test 39 and this 
time is used, along with variability in item elapsed times, to assess the user's time management skills. 
[0046] According to one embodiment of the invention, the program calculates the Difficulty Level ("DL") scores for 
each sub-section of the test 37. DL scores are calculated by using the difficulty level code for each item for which the 

40 user made an incorrect response choice. For example, on the SAT examination, there are three item difficulty levels - 
easy, moderate and difficult - each containing approximately one-third of the items in a given sub-section. One imple- 
mentation of differential weighting is to assign the values of 5, 3, and 1 to these levels respectively. Hence, an incorrect 
response to three easy items yields a DL score of 15, while an incorrect response to three difficult items yields a DL 
score of 3. 

45 [0047] This differential weighting of incorrect items assumes that users who miss easy items are in greater need of 
remediation and/or skill development than are those users that correctly answer the easy items and primarily miss the 
more difficult ones. The difficulty levels shown in this embodiment of the invention are given by way of illustration. 
Greater or fewer numbers of difficulty levels may be assigned and the method of weighting such levels may vary. 
Further, the calculation of overall DL scores may be made using a different algorithm than that shown here yet still 

50 remain within the scope of the invention. 

[0048] According to an embodiment where constrained open-ended items are employed which consist of the stem 
portions of analogous MC items, the item difficulty level information that is available for the MC items is also available 
for the constrained open-ended items, although the calculation of the overall DL scores for the constrained open-ended 
items is made using a different algorithm. The overall DL scores for the MO items and the constrained open-ended 

55 items can also differ as a consequence of differential weightings and or combinatorial algorithms being employed. 
[0049] According to anotherembodiment of the invention, the program calculates and sums the Response Alternative 
("RA") scores across all of the items in each sub-section of the test 38. RA's are codes that are assigned to each 
response alternative and are indicative of a particular distractortype. RA scores permit the determination of user sen- 
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sitivity to different distractors which, by definition, constitute wrong choices. Correct alternatives may also receive a 
code since they also may be a distractor or incorrect choice when used in a different exam question. According to one 
alternative to this embodiment, RA scores are implemented as the frequency with which a particular code is chosen 
across all items within a test sub-section. According to an alternative embodiment, a weighting scheme or another 
5 alternate scoring method known to those of ordinary skill in the field of the invention may be used to assess the RA 
scores. 

[0050] According to an embodiment where constrained open-ended items are employed which consist of the stem 
portions of analogous MC items, the item information that is available for the MC items is also available for the con- 
strained open-ended items. In this regard, the RA codes that are applicable to the MC items are also applicable to the 
10 constrained open-ended items. The set of analogous MC RA codes is expandable in its range to encompass the greater 
variability of responses to constrained open-ended items. Thus, while the MC items may have one correct response 
RA code and four incorrect alternative RA codes, the constrained open-ended items will have at least one correct 
response RA code and multiple incorrect alternative RA codes, which codes are applicable to a range of response 
values. 

15 [0051] By way of example, several subtraction problems can be considered: (1) 21-10, (2) 21-11, and (3) 21-12. 
Students learning two place subtraction may encounter difficulty when the problem requires manipulation of the tens 
and units values (i.e., "borrowing", as when the problem becomes 21 take away 12). According to one embodiment of 
the invention, a student who displays a pattern of correct responses to test items that do not require borrowing (e.g., 
21 take away 1 1 ) and incorrect responses to those test items that do, is preliminarily diagnosed as having an incorrect 

20 or non-existent model of borrowing. According to this embodiment, a more accurate diagnosis of the two-place sub- 
traction error is made upon subsequent testing focused specifically on that problem. The reliability of this diagnosis is 
partially dependent upon the number of items in the assessment. According to this embodiment, the two-place sub- 
traction problem error is diagnosed by presenting the user with test items designed to reveal this error, and once this 
error is diagnosed, the examinee is instructed in the concept of borrowing, and the examinee is presented with practice 

25 items that exercise this particular skill. 

[0052] Several additional questions are presented below byway of example to demonstrate the analysis of RA scores 
across a variety of disciplines prior to the assignment of RA codes: 

1 . Sentence Completion (a.k.a., Fill in the Blank) 

30 

[0053] Sample Question : The psychologist set up the experiment to test the rat's ; he wished to 

see how well the rat adjusted to the changing conditions it had to face. 

[0054] A user understanding the question would search for an answer that is logically related to the concept of 
"adjustment to change". 

35 

(1) Incorrect Answer A: reflexes 

RA analysis: This answer is erroneously related to "test" as in "test of reflexes"; this answer is not logically 
related to adjustment to change; 

(2) Incorrect Answer B: communicability 

40 RA analysis: This answer is also not logically related to adjustment to change; 

(3) Incorrect Answer C: stamina 

RA analysis: This answer is erroneously related to "see how well"; It is not logically related to adjustment to 
change; 

(4) Incorrect Answer D: sociability 

45 RA analysis: This answer is not logically related to adjustment to change; 

(5) Correct Answer E: adaptability 

RA analysis: This answer means adjustment to change. 

[0055] According to an embodiment where a constrained open-ended item is employed consisting of the stem portion 
50 of an analogous MC item, the item would read the same and the examinee would be requested to enter an appropriate 
word(s) in the blank. Since the stem for the constrained open-ended item is the same as that for the MC format item, 
the same set of response alternatives is applicable. An expanded range of response alternatives can be employed for 
the open-ended items so as to permit the use of synonyms. The RA analysis of the user-generated responses would 
parallel that of the MC items. Incorrect distractors for the MC items, if given in response to the constrained open-ended 
55 items, would be assigned the same codes. Incorrect distractor synonyms would also be assigned the same codes as 
the initial distractor. 
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2. Analogies 

[0056] Sample Question: FISH: TROUT as : 

[0057] A user understanding this question would searcli for an answer in wliich the first itenn represents a class and 
5 the second itenn represents a member of the specified class. 



(1) Incorrect Answer A: Ocean: Wave 

RA analysis: This answer illustrates 

(2) Correct Answer B: Mammal: Whale 

RA analysis: This answer illustrates 

(3) Incorrect Answer 0: Bird: Aviary 

RA analysis: This answer illustrates 

(4) Incorrect Answer D: Antenna: Insect 

RA analysis: This answer illustrates 

(5) Incorrect Answer E: Stag: Doe 

RA analysis: This answer illustrates 



a part to a whole; 

a class and a member; 

a defining characteristic of an item; 

a part to a whole; 

a gender and its antonym. 



3. Mathematics 



20 [0058] Sample Question: If (2+x) / (5+x) = 2/5 + 2/5, then x = ? 

[0059] A user understanding this question would search for an answer obtained by means of addition and simplifi- 
cation of the terms. 



(1) Incorrect Answer A: 2/5 

25 RA analysis: The answer is a simple eye-catcher option which is faulty; 

(2) Incorrect Answer B: 1 

RA analysis: The answer was not obtained through addition and simplification; 

(3) Incorrect Answer 0: 2 

RA analysis: The answer was obtained through repetition of a number in the stem, generally a wrong choice; 
30 (4) Incorrect Answer D: 5 

RA analysis: The answer was obtained through repetition of a number in the stem, generally a wrong choice; 
(5) Correct Answer E: 1 0 

RA analysis: The answer was obtained through addition and simplification. 



35 [0060] According to an embodiment of the invention as illustrated in Figs. 6 and 7, the program calculates the DL 
scores 37, RA scores 38, and averaged elapsed response time 39 for sub-sections "s" to "t" completed by the user. 
The RA scores and DL scores are separately rank-ordered 40 for each sub-section "s" of the sample diagnostic test. 
The program determines the relative rankings of the RA and DL scores 41 . If both are high on the given scale, indicating 
that the user incorrectly responded to many "easy" items using several incorrect response alternatives, then a prelim- 

40 inary diagnosis 42, 45 is made to remediate both content and awareness of distractor types. If the RA scores are high 
and the DL scores are moderate on the given scale, then a preliminary diagnosis 43, 46 of moderate levels of reme- 
diation of both content and awareness of distractor types is made. Finally, if RA scores are moderate to high, and DL 
scores are low on the given scale, then a preliminary diagnosis 44, 47 is made of minimal content and distractor 
difficulty; a diagnosis which is expected from users obtaining high test scores. 

45 [0061] The preliminary diagnostic recommendations 45-47 are based on data that may be displayed in a two-dimen- 
sional frequency chart - item by response alternative types with the items being grouped by difficulty level. An example 
of such a chart is shown in Table 1 below: 



TABLE 1 



50 



Item Difficulty 


RA Codes 


Wt. 


ltem# 


1 


2 


3 


4 


5 


6 




DLS 




1 


1 
















5 

(Easy) 


2 


1 














15 




3 






1 
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TABLE 1 (continued) 



Item Difficulty 


RA Codes 


Wt. 


ltem# 


1 


2 


3 


4 


5 


6 




DLS 




4 


















Q 

o 

(Mid) 


cr 
O 
















O 




6 












1 








7 






1 












1 

(Hard) 


8 


1 






1 








3 




9 


















Sum 


RA 


11 


0 


6 


4 


0 


3 







[0062] The chart above indicates that the user chose response alternative coded 1 for itenn number 1 , response 

alternative coded 1 for item number 2, response alternative coded 3 for item number 3, and so on. For each of the 
three items, the choices made were incorrect. Note that the correct alternative code for each of the items is not indicated 
in the chart and according to this embodiment, scoring is based only on incorrect responses selected. Alternatively, 
the correct answers could also be ranked on the chart shown in Table 1 . Furthermore, while only one method of ranking 
is presented above, many others methods of ranking can be used, as would be understood by one skilled in the field 
of the invention. According to an embodiment where constrained open-ended items are employed which consist of the 
stem portions of analogous MC items, the difficulty levels of the stem-equivalent items will be equivalent to those 
assigned to the MC items and the constrained open-ended items will share an overlapping set of RAs. However, the 
constrained open-ended items will be assigned a larger number of RAs so as to encompass a larger range of user- 
generated responses. 

[0063] Figs. 8 and 9 are block diagrams of the remediation component and the test generation component, and 
illustrate the interaction between the two components according to another embodiment of the invention. According to 
this embodiment, a preliminary diagnosis is confirmed by presenting the user with a subsequent quiz following a diag- 
nostic test. For example, where a user has responded in a particular way to a set of analogies on the diagnostic test 
using only one or two distractortype codes (analogy type codes) then a quiz of analogy type codes is offered to develop 
a firm and reliable diagnosis. Following the quiz, an appropriate remedial training exercise presents the user with a 
set of analogies in which the various types of analogies are made explicit. Such exercises increase the user's knowledge 
of various types of analogical relationships. 

[0064] According to another embodiment of the invention, after the preliminary diagnosis and recommendations are 
made 42-47, the preliminary diagnostic recommendations are combined with data from the demographic questionnaire 
4, the stored response alternative codes 50, the stored infomnation with regard to content to the applicable exam 51 , 
and test-taking behavior (e.g., elapsed times per sub-section) to determine RA codes and/or areas of content that 
require further assessment 48. If data from a sample diagnostic test is not available, as in the case of a user who wants 
to access only the skill development portion of the program, then no diagnostic recommendations are available to 
structure the formation of remediation quizzes and/or exercises 49. In this case, content quizzes and exercises are 
structured using a pre-set format until such user data becomes available. 

[0065] For example, if the RA scores for the analogies sub-section suggests an insensitivity to a number of types of 
analogies, a request is passed to the Test Generation component 1 0, 52 for a quiz 53 to assess the user's knowledge 
of types of relationships commonly employed in standardized tests. According to one embodiment, such a quiz 53 is 
selected from one of a number of predetermined quizzes, each of which is designed to test a commonly encountered 
deficiency in responding to analogy problems. According to another embodiment, a quiz 53 is generated by the Test 
Generation component 1 0 in responsetothe results from the sample diagnostic test and the demographic questionnaire 
4. If the user obtains a less than satisfactory score 55, then the Test Generation Component 52 generates exercises 
appropriate to the user's score level 57. 

[0066] According to the embodiment, where a user's deficiency is detected in his understanding of the contents of 
the tested material, an analogous process is employed to provide remediation with regard to the necessary content. 
If the preliminary diagnostic recommendation and applied weighting factors 48 indicate, for example, that a user is 
manifesting considerable difficulty with content (e.g., high DL scores across numerous sub-sections of a test), and 
appears to be susceptible to particular RA codes (e.g., codes in the specific content area of algebra) then a request 
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is passed to the Test Generation Connponent 52 for a quiz to assess the user's level of knowledge in that specific 
content area. If a lower than satisfactory score Is obtained on the quiz 56, then the Test Generation Component 52 
generates exercises In the applicable content area appropriate to that score level 58. This dynamic process of recom- 
mendation and remediation permits the sequencing of content presentation according to the difficulty level most likely 

5 to Increase the user's skill level. 

[0067] Additionally, preliminary diagnostic recommendations for skill development in testwiseness (i.e., skills which 
promote higher test scores which are independent of knowledge of the content areas being tested) and/or test taking 
strategy (e.g., knowledge of how to approach a test, how long to take on each question, how to deconstruct items from 
varied content domains and respond to different Item formats) are made by the program 48 and quizzes and exercises 

10 on testwiseness and test taking strategy are generated by the Test Generation Component 10, 52 according to one 
embodiment of the invention. Diagnosis of errors in test-taking strategy or testwiseness are based upon logical grounds 
and the frequency of occurrence. By way of illustration, if easy items are skipped and difficult Items are attempted, a 
recommendation regarding a more appropriate sequence is made. Similarly, if the user's target score does not require 
responses to difficult items, but such items are attempted, a recommendation is made regarding the order in which 

15 items should be attempted. 

[0068] Once the appropriate questions are formatted by the Test Generation Component 1 0, 52 for knowledge and 
skill development In a particular content area, with respect to particular distractor identification and awareness, or with 
respect to testwiseness and test taking strategy, quiz items are displayed in multiple choice format with various response 
alternatives 59 and the user may select from among the choices 60. If the questions are being presented in a quiz 

20 mode, no feedback Is displayed until the entire quiz has been completed 61 . If the questions are being presented In a 
training or skill development exercise mode, feedback may be displayed in a variety of user-selected modes, varying 
by level, timing, and provision of solution and/or answer. 

[0069] According to one embodiment, the variety of hint levels ranges from minimal, in which a general statement 
regarding problem types Is provided, to moderate, In which words, phrases, and/or numbers may be highlighted, to 
25 maximum, in which the entire solution is provided. According to another embodiment, problems solution Is provided 
on demand or automatically after an incorrect option choice. Additionally, answers may be provided after each item, 
after each sub-section or after the entire test is completed. Solutions may also be provided after each item, after 
incorrect responses only, after a correct response if a better option was available, or at the end of test sub-section. By 
way of illustration, an analogy exercise might read as follows: 

30 

Item Presented: FISH:TROUT 

A. OCEAN:WAVE 

B. MAMMAL:WHALE 
35 C. BIRD:AVIARY 

D. ANTENNAL:INSECT 

E. STAG:DOE 

[0070] A minimal hint may be the statement: "Consider the relationship between the capitalized words and each of 
40 the words in the response alternatives." A moderate level hint may read: "Make a sentence such as "trout Is a kind of 
fish" and substitute response alternative words into the sentence, remembering to reverse the order. If the result Is 
nonsense, then eliminate that alternative." A maximum hint may be: "B. MAMMAL: WHALE". 

[0071] The program loops until all questions in a quiz or exercise are presented 62, at which point the program 
displays a variety of summary feedback (e.g., score, areas of excellence and improvement) 63. Once all the items are 
45 presented, the user is presented with an option menu 64 that provides such choices as additional remediation or skill 
development 47a, additional sample diagnostic tests 11 , or the ability to exit the program. 

[0072] The User Interface Component 1 also manages a number of other options for personalizing the program for 
particular users. According to one embodiment, instructional materials are presented in multi-media style so as to 
optimize instruction for different user learning styles. The materials can also be presented in different font sizes, with 
50 different cursor styles. In different colors. In different locations on the screen, and with different types of music or audio 
accompaniment. The User Interface Component may also provide difficulty meter levels for illustrating the difficulty 
levels of given problems, arrow buttons for scrolling between problems, bookmarking such that a user can bookmark 
and return to particular Items later In an exam, and an elapsed time display and pacer clock. 

[0073] One skilled in the art will appreciate that the foregoing embodiments are presented for purposes of illustration 
55 and not of limitation. 
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Claims 

1 . A computerized method for preparing a user for standardized examinations, said metliod comprising tlie steps of: 

5 (a) generating a diagnostic test for assessing the user's knowledge of a domain being tested, said test con- 

taining a plurality of questions, each question having a corresponding answer set, each said answer set pos- 
sessing at least one correct answer and at least one incorrect answer, and one or more of said answers being 
respectively associated with one or more distractor error codes; 
(b) presenting a series of said questions to said user; 

10 (c) eliciting one of said answers from said user for each said question; 

(d) recording said user's elicited answer for each said question; 

(e) analyzing said distractor error codes associated with the elicited answer; 

(f) detecting one or more distractor error patterns from said analysis; 

(g) diagnosing a deficiency in said domain being tested based on said distractor error patterns; and 
15 (h) generating a remediation activity to remediate said diagnosed deficiency. 

2. The method of claim 1 , wherein said questions are constrained open-ended questions. 

3. The method of claim 1 , wherein: 

20 

step (b) further comprises presenting said answer set associated with each said question to said user for 
selection by said user of one of said answers in said answer set. 

4. The method of claim 1 , further comprising presenting said remediation activity to said user, and tracking said user's 
25 performance with respect to said presented remediation activity. 

5. The method of claim 1 , further comprising soliciting demographic information from said user, and generating said 
remediation activity based on said demographic information. 

30 6. The method of claim 1 , wherein: 

step (a) further comprises assigning said one or more distractor error codes to identify a deficiency in test- 
wiseness skills of said user; 

step (e) further comprises analyzing said user's selected answers to identify said deficiency in testwiseness 
35 skills based on the distractor error codes associated with said selected answers; and 

step (h) further comprises generating a remediation activity to remediate said deficiency based on said anal- 
ysis. 

7. The method of claim 1 , wherein said diagnosis step (g) further comprises: 

40 

(i) generating at least one additional test to confirm said diagnosis; 

(ii) presenting said additional test to said user; 

(iii) analyzing said user's responses to said additional test; and 

(iv) confirming said diagnosis. 

45 

8. The method of claim 1 , further comprising (i) providing feedback to said user with respect to said remediation 
activity. 

9. The method of claim 1 , wherein said questions include questions taken from a standardized achievement test. 

50 

10. The method of claim 1 , wherein said distractor error codes include distractor error codes that indicate one or more 
types of errors in reasoning. 

11. The method of claim 1 , wherein said step of generating said remediation activity (h) further comprises analyzing 
55 said user's performance on previous remediation activities. 

12. A computer readable media containing computer instructions for preparing a user for standardized examinations, 
said computer readable media comprising: 
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(a) a test generating computer code for generating a diagnostic test for assessing said user's knowledge with 
respect to a donnain of knowledge being tested, said test containing a plurality of questions, eacli question 
liaving a corresponding answer set, eacli said answer set possessing at least one correct answer and at least 
one incorrect answer, and one or more of said answers being respectively associated with one or more dis- 

5 tractor error codes; 

(b) a test display computer program code for displaying a series of said questions to said user; 

(c) an eliciting computer program code for eliciting one of said answers from said user for each said question; 

(d) a recording computer code for recording said user's elicited answer to each said question; 

(e) an analysis computer program code for analyzing the distractor error codes associated with the elicited 
10 answer; 

(f) a detecting computer program code for detecting one or more distractor error patterns from said analysis; 

(g) a diagnosis computer program code for diagnosing a deficiency in said domain being tested based on said 

distractor error pattern; and 

(h) a remediation generator computer program code for generating a remediation activity to remediate said 
15 deficiency. 

13. The computer readable media of claim 12, wherein said questions are constrained open-ended questions. 

14. The computer readable media of claim 12, wherein: 

20 

said test display computer code (b) further comprises a computer code for presenting said answer set asso- 
ciated with each said question to said user for selection by said user of one of said answers in said answer set. 

15. The computer readable media of claim 12, further comprising: 

25 

(i) an interface computer code for managing said user's interaction with said media. 

16. The computer readable media of claim 12, further comprising: 

30 (j) a presentation computer code for presenting said remediation activity to said user for completion and for 

tracking said user's performance with respect to said presented remediation activity. 

17. The computer readable media of claim 16, wherein said presentation computer code (j) further comprises: 

35 (k) a user selection computer code for presenting said remediation activity to said user in a manner selected 

by said user. 

18. The computer readable media of claim 12, further comprising: 

40 (I) a demographic computer code for soliciting demographic information from said user; and 

(m) a remediation tailoring computer code for generating said remediation activity based on said demographic 
information. 

19. The computer readable media of claim 12, wherein said distractor error codes include error codes for identifying 
45 a deficiency in testwiseness skills of said user, and wherein said remediation generator computer code (h) further 

comprises: 

(n) a second analysis computer code for analyzing said user's selected answers to identify said deficiency in 
testwiseness skills based on the distractor error codes and test taking behavior indicators associated with said 
50 selected answers; and 

(o) a second remediation generator computer code for generating a remediation activity to remediate said 
deficiency based on said analysis. 

20. The computer readable media of claim 1 2, wherein said diagnosis computer program code (g) further comprises: 

55 

(p) an assessment test generating computer program code for generating at least one additional test to confirm 
said diagnosis; 

(q) an assessment test presentation computer code for presenting said additional test to said user to obtain 
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said user's response to said additional test; 

(r) an assessnnent analysis connputer program code for analyzing said user's response to said additional test; 
and 

(s) a confirnnation computer program code for making a confirmed diagnosis based on said analysis by said 
5 assessment analysis computer code. 

21. The computer readable media of claim 12, further comprising: 

(t) a feedbacl< computer code providing feedback to said user with respect to said remediation activity. 

10 

22. The computer readable media of claim 12, wherein said questions include questions taken from a test composed 
of standard answers. 

23. The computer readable media of claim 12, wherein said distractor error codes include distractor error codes that 
15 indicate one or more types of errors in reasoning. 

24. The computer readable media of claim 12, wherein said remediation generator computer code (h) generates a 
remediation activity based said user's performance on a previous remediation activity. 

20 25. A diagnostic testing and remediation apparatus, said apparatus comprising: 

(a) a computing device 

(b) an input device for receiving input from a user and connected to said computing device; 

(c) an output device connected to said computing device; 

25 (d) a memory storage device connected to said computing device, said memory storage device further com- 

prising: 

(i) a memory for storing a pool of test questions for assessing said user's knowledge of a domain being 
tested, each question having a corresponding answer set, each said answer set possessing at least one 

30 correct answer and at least one incorrect answer, and one or more of said answers being respectively 

associated with one or more distractor error codes; 

(ii) a test generating computer program code for generating a diagnostic test from said pool of test ques- 
tions, said diagnostic test comprising a test question subset of said pool of test questions; 

(iii) a test display computer program code for displaying said diagnostic test to said user; 

35 (iv) an eliciting computer program code for eliciting one of said answers from said user for each said 

question in said diagnostic test; 

(v) a recording computer program code for recording said user's elicited answers to each said question; 

(vi) an analysis computer program code for analyzing the distractor error codes associated with said user's 

elicited answers; 

40 (vii) a detecting computer program code for detecting one or more distractor error patterns from said 

analyzed distractor error codes; 

(viii) a diagnosis computer program code for diagnosing a deficiency in said domain being tested based 
on said detected distractor error pattern; and 

(ix) a remediation generator computer program code for generating a remediation activity to remediate 
45 said deficiency. 

26. The apparatus of claim 25, wherein said questions are constrained open-ended questions. 

27. The apparatus of claim 25, wherein: 

50 

said test display computer code (iii) further comprises a computer code for presenting said answer set asso- 
ciated with each said question to said user for selection by said user of one of said answers in said answer set. 

28. The apparatus of claim 25, wherein said computing device is located locally to said user. 

55 

29. The apparatus of claim 25, further comprising a communication link connecting said input device and said output 
device with said computing device, and wherein said computing device is a computer located remotely from said 
user. 
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30. The apparatus as recited in ciaiiri 25, wherein said nriennory storage device (d) further comprises: 

(x) an interface computer code for managing said user's interaction with said apparatus. 
5 31 . The apparatus of claim 25, wherein said memory storage device (d) further comprises: 

(xi) a demographic computer code for soliciting demographic information from said user; 

(xii) a second memory for storing said demographic information; and 

(xiii) a remediation tailoring computer code for generating said remediation activity based on said demographic 
10 information. 

32. The apparatus of claim 25, wherein said memory storage device (d) further comprises: 

(xiv) a presentation computer code for presenting said remediation activity to said user for completion and for 
15 tracking said user's performance with respect to said remediation activity. 

33. The apparatus of claim 25, wherein said memory storage device (d) further comprises: 

(xv) a user selection computer code for presenting said remediation activity to said user in a manner selected 
20 by said user. 

34. The apparatus of claim 25, wherein said memory storage device further comprises: 

(xvi) a feedback computer code for providing feedback to said user with respect to said remediation activity. 

25 

35. The apparatus of claim 25, wherein said test display computer code (iii) further comprises: 

(xvii) a timer display computer code for displaying a timer. 
30 36. The apparatus of claim 35, further comprising: 

(xviii) a second timer computer code for calculating an elapsed time in connection with said user's selected 
answers to said questions and for displaying said elapsed time. 

55 37. The apparatus of claim 25, wherein said pool of test questions includes questions taken from a standardized 
achievement test. 

38. The apparatus of claim 25, wherein said distractor error codes include distractor error codes that indicate one or 
more types of errors in reasoning. 

40 

39. The apparatus of claim 25, wherein said distractor error codes include error codes for identifying a deficiency in 
testwiseness skills of said user, and wherein said remediation generator computer code (ix) further comprises: 

(xix) a second analysis computer code for analyzing said user's selected answers to identify said deficiency 
45 in testwiseness skills based on the distractor error codes associated with said selected answers; and 

(xx) a second remediation generator computer code for generating a remediation activity to remediate said 
deficiency based on said analysis. 

40. The apparatus of claim 25, wherein said diagnosis computer code (viii) further comprises: 

50 

(xxi) an assessment test generating computer program code for generating at least one additional test to 

confirm said diagnosis; 

(xxii) an assessment test presentation computer code for presenting said additional test to said user to obtain 
said user's response to said additional tests; 

55 (xxiii) an assessment analysis computer program code for analyzing said user's response to said additional 

test; and 

(xxiv) a confirmation computer program code for making a confirmed diagnosis based on said analysis by said 
assessment analysis computer code. 
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A computerized standardized test preparation apparatus, said apparatus connprising: 

(a) means for generating a diagnostic test for assessing a user's knowledge of a domain being tested, said 
test containing a plurality of questions, each question having a corresponding answer set, each said answer 
set possessing at least one correct answer and at least one incorrect answer, and one or more of said answers 
being respectively associated with one or more distractor error codes; 

(b) means for presenting a series of said questions to the user; 

(c) means for eliciting one of said answers from said user for each said question; 

(d) means for recording said user's selected answer to each said question; 

(e) means for analyzing said distractor error codes associated with the elicited answer; 

(f) means for detecting one or more distractor error patterns based on said analyzed distractor error codes; 

(g) means for diagnosing a deficiency based on said error pattern in said domain being tested based on said 
distractor error patterns; and 

(h) means for generating a remediation activity to remediate said deficiency. 
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ABSTRACT: 

CHG DATE=20021101 STATUS=0> A computerized test 
preparation system and methods are disclosed which 
assist an examinee to increase his learning and 
improve his performance on standardized academic 
or applied aptitude and achievement exams. 
Performance feedback information is provided to a 
user, including conventional information such as 
number of items correct and time elapsed, 
pinpoints substantive areas of a particular exam 
in which the user is either weak or strong, and 
offers diagnoses of, and remediation tailored to, 
a user ' s personal difficulties . By assessing a 
user's responses to given exam questions, through 
various distractor error patterns manifested in 
the incorrect answers to multiple-choice 
questions, or through coded categories of 
responses in the case of user responses to 
constrained open-ended items, a preliminary 
diagnosis of a user's deficits in knowledge and/or 
test taking skill is made. This preliminary 
diagnosis is subsequently refined utilizing 
additional examination questions. The refined 
diagnosis is then utilized to offer 
recommendations to an individual user to 
ameliorate the user's manifested error patterns. A 
user's response to this remediation is confirmed 
and reinforced using subsequent remediation 

activities. 1 1 
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